On the Whittle index of Markov modulated restless bandits

In this paper, we study a Multi-Armed Restless Bandit Problem (MARBP) subject to time fluctuations. This model has numerous applications in practice, like cloud computing systems or wireless communications networks. Each bandit is formed by two processes: controllable process and an environment. The transition rates of the are determined state environment, which exogenous Markov process. decision maker full information on every bandit, objective determine optimal policy that minimises long-run average cost. Given complexity problem, set out characterise Whittle index, obtained solving relaxed version MARBP. As reported literature, heuristic performs extremely well for wide variety problems. Assuming problem threshold type, provide algorithm finds Whittle’s index. We then consider multi-class queue with linear cost impatient customers. For model, show optimality, prove indexability, obtain index closed-form. also limiting regimes environment relatively slower faster than By numerical simulations, assess suboptimality scenarios, general observation that, as case standard MARBP, gap small.